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A METHOD AND APPARATUS FOR ENHANCING THE BROADCAST 

OF AUVB EVENT 



RACKGROIIND OF TH E INVENTTON 
Field of the Invention 

The present invention is directed to a method and apparatus for 
enhancing a television broadcast of a live event. 

Description of the Related Art 

The television presentation of live events could be improved by 
enhancing the video in real time to make the presentation more interesting to the 
viewer. For example, television viewers cannot see the entire playing field 
during a sporting event; therefore, the viewer may lose perspective as to where 
one of the players or objects are on the field in relation to the rest of the field, 
players or objects. During the telecast of football games cameras tend to zoom 
in on the players which allows the viewer to only see a small portion of the 
field. Because the viewer can only see a small portion of the field a viewer may 
not know where a paiticnlar player is in relation to the pertinent locations on the 
field. One instance is when a pkyer is carrying the football, the television 
viewer may not know how far that player has to run for a first down. One 
enhancement that would be helpful to television viewers of football games is to 
highlight the field at the point where a player must advance in order to obt^n 
a first down. 

An enhancement that would be helpful to viewers of golf tournaments 
is to highlight those portions of a golf course that have been notorious trouble 
spots to golfers. While the professional golfer is aware of these trouble spots 
and hits the ball to avoid those spots, the television viewer may not be aware 
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of those trouble spots and may wonder why a particular golfer is hitting the ball 
in a certain direction. If the golf course was highlighted to show these trouble 
spots, a television viewer would understand the strategy that the golfer is using 
and get more enjoyment out of viewing the golf tournament. Another useful 
5 enhancement would iuclude showing the contours of the green. Similar 
enhancements to the playing field would be usefurin other sports as well. 

Furthermore, live events do not take advantage of the scope of the 
television audience with respect to advertising. First, advertisements on display 
at a stadium Can be televised; however, many of those advertisements are not 
10 applicable to the television audience. For example, a particular sporting event 
may be played in San Francisco and televised around the world. A local store 
may pay for a billboard at the stadium. However, viewers in other parts of the 
United States or in other countries receiving the broadcast may not have access 
to that store and, thus, the broadcast of the advertisement is not effective. 
15 Second, some of the space at a stadium is not used because such use would 
interfere with the view of the players or the spectators at the stadium. 
However, using that space for advertisement would be very effective for the 
television audience. For example, the glass around the perimeter of a hockey 
rink would provide an effective place for advertisements to the television 
20 audience. However, such advertisements would block the view of spectators 
at the stadium. Third, some advertisements would be more effective if their 
exposure is limited to particular times when customers are thinking of that type 
of product. For example, an advertisement for an umbrella would be more 
effective while it was raining . 
25 Previous attempts to enhance the video presentation of live events have 

not been satisfactory. Some broadcasters superimpose advertisements on the 
screen; however, these advertisements tend to block the view of the event. 
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Another solution included digitizing a frame of video and using a 
computer with pattern recognition software to locate the target image to be 
rq)laced in the frame of video. When the target image is found, a replacement 
image is inserted in its place. The problem with this solution is that the 

5 software is too slow and cannot be effectively used in conjunction with a live 
event. Such systems are even slower when they account for occlusions. An 
occlusion is something that blocks the target. For example, if the target is a 
billboard on the boards around a hockey rink, one example of an occlusion is 
a player standing in front of the billboard. When that billboard is replaced, the 

10 new billboard image must be inserted into the video such that the player appears 
to be in front of the replacement billboard. 

SUMMARY OF THE INVENTION 
The present invention is directed to a system for enhancing the broadcast 
15 of a live event. A target, at a hve evMt, is selected to be enhanced. Examples 
of targets include advertisements at a stadium, portions of the playing field 
(e.g., football field, baseball field, soccer field, basketball court, etc.), 
locations at or near the stadium, or a monochrome background (e.g. for 
chroma-key) positioned at or near the stadium. The system of the present 
20 invention, roughly described, captures video using a camera, senses field of 
view data for that camera, determines a position and orientation of a video 
image of the target in the captured video and modifies the captured video by 
enhancing at least a portion of the video image of the target. Alternative 
embodiments of the present invention include determining the perspective of the 
25 video image of the target and/or preparing an occlusion for the video image of 
the target. 

One embodiment of the present invention includes one or more field of 
view sensors coupled to a camera such that the sensors can detect data from 
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which the field of view of the camera can be determined. The field of view 
sensors could include pan, tilt and/or zoom sensors. The system also includes 
a processor, a memory and a video modification unit. The memory stores a 
location of the target and, optionally, data representing at least a portion of the 

5 video image of the target. The processor, which is in communication with the 
memory and the field of view sensors, is programmed to determine whether the 
target is within the field of view of the camera and, if so, the position of the 
target within a frame of video of the camera* Alternate embodiments allow for 
the processor to determine the position of the target in the frame of video using 

10 field of view data, pattern (or image) recognition technology, electromagnetic 
signals and/or other appropriate means. One exemplar embodiment uses field 
of view data to find a rough location of the target and then uses pattern 
recognition to find the exact location. Such a combination of field of view data 
with pattern recognition technology provides for faster resolution of the target' s 

15 location than using pattern recognition alone. 

The video modification unit, which is in communication with the 
processor, modifies the frame of video to enhance at least a portion of the video 
image of the target. That is, a target can be edited, highlighted, overlayed or 
rq)laced with a replacement image. For example, a video modification unit can 

20 be used to highlight a portion of a football field (or other playing field) or 
replace a first billboard in a stadium with a second billboard. Because the 
system can be configured to use pattern recognition technology and field of 
view sensors, the system can be used with multiple broadcast cameras 
simultaneously. Therefore, a producer of a live event is free to switch between 

25 the various broadcast cameras at the stadium and the television viewer will see 
the enhancement regardless of which camera is selected by the producer. 

An alternate embodiment contemplates rq)lacing either the field of view 
sensors and/or the pattern recognition technology with electromagnetic 
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transmitters and sensors. That is, the target can be used to emit an 
electromagnetic signal* A sensor can be placed at the camera, or the camera 
can be used as a sensor, to detect the signal from the target in order to locate 
the target. Once the target is located within the video frame, the system can 

5 enhance the video image of the target. A fiirther alternative includes treating 
the target with spectral coatings so that the target wiU reflect (or emit) a distinct 
signal which can be detected by a camera with a filter or other sensor. 

These and other objects and advantages of the invention wiU appear 
more clearly from the following description in which the preferred embodiment 

10 of the invention has been set forth in conjunction with the drawings. 

BRIEF DRSCRIPnON OP THE DRAWINGS 
Figure 1 depicts a jjerspective view of part of a football stadium. 
Figure 2 dq)icts a i)erspective view of the football stadium of Figure 1 
15 as seen by a television viewer after the video has been enhanced. 

Figure 3 depicts a block diagram of a subset of the components that 
make up the present invention. 

Figure 4 depicts a block diagram of a subset of the components that 
make up the present invention. 
20 Figure 5 is a flow chart describing the operation of the present 

invention. 

Figure 6 is a flow chart which provides more detail of how the present 
invention accounts for occlusions. 

Figure 7 is a partial block diagram of an alternate embodiment of the 
25 present invention* 

Figure 8 is a partial flow chart describing the operation of the alternate 
embodiment depicted in Figure 7. 
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P ETMX.ED PR SCRIFnON 
Figure 1 is a partial view of football stadium 100, In the center of 
stadium 100 is a football field 102, Surrounding football field 102 are the seats 
104 for the fans. Between seats 104 and playing field 102 is a retaining wall 
5 106. On retaioing wall 106 is an advertisement ADl, For example purposes 
only, assume that a particular television broadcaster has selected four targets for 
enhancement. The first target is an advertisement ADl to be replaced by 
another advertisement. The second target is a portion of the playing field which 
is to receive an advertisement. For this example, assume that the broadcaster 
10 wishes to place an advertisement in the end zone 108 of the football field. A 
third target is an area above the stadium. That is, the television broadcaster 
may wish that when a camera is pointed to the top of the stadium, the viewers 
sees an advertisement suspended above the stadium. A fourth target is a 
location on the playing field 102 representing where a team must cross in order 
to get a first down. Although the television broadcaster may be enhancing the 
video image as discussed above, the spectators and players at the stadium would 
not see any- of these enhancements, rather they would view the stadium as 
depicted in Figure 1. 

Figure 2 shows the view of Figure 1, as seen by viewers watching the 
20 broadcast on television, after enhancements are made to the video. 
Advertisement AD2 is in the same location as advertisement ADl was in 
Figure 1. Thus, advertisement AD2 has replaced advertisement ADl. 
Advertisement ADS is shown in end zone 108. Advertisement AD3 does not 
xeplsLCQ another advertisement because there was no advertisement in end zone 
25 108 prior to the enhancement. Figure 2 also shows advertisement AD4, which 
to the television viewer appears to be suspended above stadium 100. Also 
shown in Figure 2 is a thick line 110 which rq)resents the highlighting of the 
portion of the field at which the team who is offense must cross in order to get 
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a first down at a particular moment during the game. In this particular 
example, the highlighting of the field consists of a bold thick line. Alternatives 
include different color lines, shading, usmg a blinking line, varying the 
brightness, etc. The enhancement need not be a line. The enhancement may 
5 also be any other shape or graphic that is appropriate. Thus, for purposes of 
this patent an enhancement includes editing an image, replacing part of an 
image with another image, overlaying all or part of an image, highlighting an 
image using any E^ropriate method of highlighting, or replacing an image with 
video. 

10 Figure 3 is a block diagram of a subset of the components that make up 

the present invention. Hie comfKJnents shown on Figure 3 are typically located 
at a camem bay in the stadium; however, they can be located in other suitable 
locations. Broadcast camera 140 captures a frame of video which is sent to a 
production center as shown by the signal BCl, Broadcast camera 140 has a 
15 zoom lens, including a 2X E?q>ander (range extender). Connected to broadcast 
camera 140 is a 2X Expander/zxK)m/focus sensor 152 (collectively a "zoom 
sensor") which seises the zoom in the camera, the focal distance of the camera 
lense, and whether the 2X Expander is being used. The analog output of sensor 
152 is sent to an analog to digital converter 154, which converts the analog 
20 signal to a digital signal, and transmits the digital signal to processor 156, One 
alternative includes using a zoom sensor with a digital output, which would 
remove the need for analog to digital converter 154. Broadcast camera 140 is 
mounted on tripod 144 which includes pan and tilt heads that enable broadcast 
camera 140 to pan and tilt. Attached to tripod 144 are pan sensor 146 and tilt 
25 sensor 148, both of which are connected to pan-tilt electronics 150. 
Alternatively, broadcast camera 140 can include a built-in pan and tilt unit. In 
either configuration, pan sensor 146, tilt sensor 148 and zoom sensor 152 are 
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considered to be coupled to broadcast camera 140 because they can sense data 
representing the pan tilt, and zoom of broadcast camera 140. 

Processor 156 is an Intel Pentium processor with supporting electronics; 
however, various other processors c^ be substituted. Processor 156 also 
includes memory and a disk drive to store data and software. In addition to 
being in communication with pan-tilt electronics 150 and analog to digital 
converter 154, processor 156 is in communication (via signal CBl) with a 
production center which is described below. 

In one embodunent, pan sensor 146 and tilt sensor 148 are optical 
encoders that output a signal, measured as a number of clicks, indicating the 
rotation of a shaft. Forty thousand (40,000) clicks represent a full 360^ 
rotation. Thus, a processor can divide the number of measured clicks by 
40,000 and multiply by 360 to determine the pan or tilt angle in degrees. The 
pan and tilt sensors use standard technology known in the art and can be 
replaced by other suitable pan and tilt sensors known by those skilled in the 
relevant art. Pan/tilt electronics 150 receives the output of pan sensor 146 and 
tilt sensor 148, converts the ou^ut to a digital signal (representing pan and tilt) 
and transmits the digital signal to processor 156. The pan, tilt and zoom 
sensors are used to determine the field of view of the broadcast camera. Thus, 
one or more of the pan, tilt or zoom sensors can be labeled as a field of view 
senor(s). For example, if a camera cannot zoom or tilt, the field of view sensor 
would only include a pan sensor. 

An alternative field of view sensor includes placmg marks in various 
known locations in the stadium such that each mark looks different and at least 
one maik will always be visible to the camera while the camera is iK>inted at the 
relevant portions of the stadium. A computer usmg pattern recognition 
technology can find the maik in a frame of video and, based on the mark's size 
and position in the frame of video, determme more precisely the field pf view 
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and/or pan, tilt or zjoom of the camera. A system can also be set up to use 
pan/tilt/zoom sensors in combination with the marics described above so that the 
pan/tat/zoom can be used to make a rough estimate of where the camera is 
pointing and the mark is used to achieve a more accurate estimate. In such a 
5 combination system the marks need not look different if the placement of the 
marks is predetermined. Another alternative includes placing infrared emitters 
or beacons along the perimeter of the^playing field or other portions of the 
stadium. A computer can determine an infiared sensor's field of view based on 
the location of the signal in the infrared sensor's frame of data. If the infrared 
10 sensor is mounted on a broadcast camera, determining the pan and tilt of the 
infiared sensor determines the pan and tilt of the broadcast camera plus a known 
offset. A more detailed discussion of using infrared technology, pan/tilt/zoom 
sensors, three dimensional location finding technology and video enhancement 
can be found in U.S. Patent Application No. 08/585,145, A System For 
15 Enhancing The Television Presentation Of An Object At A Sporting Event, 
incorporated herein by reference. 

Figure 3 shows a second and optional camera labeled as dedicated 
camera 142. Dedicated camera 142 is mounted on a tripod 157. In one 
embodiment, trqxd 157 includes an optional pan sensor 158 and an optional tilt 
20 sensor 160, both of which are in communication with pan-tilt electronics 150. 
As will be explained below, in one embodiment the dedicated camera is set to 
one pan and tilt position; therefore, pan and tUt sensors are not needed. The 
output of dedicated camera 142 is the camera signal DCl, which is 
communicated to the production center described below. The present invention 
25 will perform its function without the use of dedicated camera 142; however, 
dedicated camera 142 improves the ability of the system to account for 
occlusions. Dedicated camera 142 should be located substantially adjacent to 
broadcast camera 140. That means that dedicated camera 142 should be as 
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close as possible to broadcast camera 140 so that both will function properly yet 
their optical axes will be as close as practical. Thus, if both cameras are 
focused on the same object, their pan and tilt angle should be very similar. In 
various alternatives, each broadcast caihera could be associated with more than 
one dedicated cameras. In order to further enhance performance, each 
broadcast camera would include a plurality of dedicated cameras, one dedicated 
camera for each potential target the broadcast camera will view. 

Figure 4 is a block diagram of the production center. Typically, the 
production center is housed in a tmck parked outside of the stadium. However, 
the production center can be at a central office or the components of the 
production center can be spread out in multiple locations. The heart of the 
production center is processor 200, The preferred processor 200 is an Onyx 
computer fix)m Silicon Graphics; however, various other suitable processors or 
combmations of processors can perform the necessary functions of the present 
invention. Processor 200 is in communication with video control 202, video 
mixer 204 and multiplexor 206. In one alternative, processor 200 includes 
more than one processor. For example, processor 200 could include two Onyx 
computers, one for locatmg the target and one for determining occlusions. 

Broadcasters use many broadcast cameras at the stadium to televise a 
sporting event. The video signals from the various cameras are sent to video 
control 202 which is used to select one broadcast camera for transmission to 
viewers. One embodiment of video control 202 includes a plurality of monitors 
(one monitor for each video signal) and a selection circuit. A director (or 
manager, producer, etc.) can monitor the different video signals and choose 
which signals to broadcast. The choice would be communicated to the selection 
circuit which selects one camera signal to broadcast. The choice is also 
communicated to processor 200, video mixer 204 and multiplexer 206 via signal 
208. The selected video signal is sent to delay 210 and processor 200 via 
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analog to digital converter 212, If the broadcast camera is a digital camera, 
then there would be no need for analog to digital converter 212. 

The output of delay 210 is sent to video modification unit 214, The 
purpose of delay 210 is to delay the broadcast video signal a fixed number of 
frames to allow time for processor 200 to receive data, determine the position 
of the target in the frame of video and prepare any enhancements, Althoughthe 
video is delayed a small number of frames, the television signal is stiU defined 
as live. The delay introduced by the system is a small delay (under one second) 
which does not accumulate. That is, different frames of video are enhanced 
with the same small delay. For example, a ten frame delay is equivalent to one- 
third of a second, which is not considered a significant delay for television. 

Video mixer 204 receives the video signals from all of the dedicated 
cameras. Figure 4 shows signals DCl and DC2. Signal DCl is a dedicated 
camera associated with the broadcast camera BCl, If video control 202 selects 
BCl then that selection is communicated to video mixer 204 which selects DCl . 
As discussed above, it is contemplated that some alternatives include having 
many dedicated cameras for one broadcast camera. For example, one broadcast 
camera may have four dedicated cameras. In that case, the dedicated cameras 
would be labeled DCla, DClb, DClc and DCld, When broadcast camera BCl 
is selected, video mixer 204 would select up to all four dedicated cameras: 
DCla, DClb, DClc and DCld. The selected signal(s) from video mbcer 204 
is sent to analog to digital converter 216 which digitizes the video signal(s) and 
sends the digital signal(s) to processor 200. 

Multiplexer 206 receives signals from the processors at each of the 
camera locations. For example, Figure 4 shows multiplexer 206 receiving 
signal CBl from processor 156 of Figure 3, Each of the processor signals 
(CBl, CB2, . , . ) is associated with a broadcast camera. Thus, the selection 
by video control 202 is communicated to multiplexer 206 so that multiplexer 
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206 can send the corresponding signal to processor 200. The signal sent by 
multiplexer 206 to processor 200 includes the information from the field of 
view sensors. In one embodiment, processor 156 calculates the field of view 
and sends the resulting information, via multiplexer 206, to processor 200. In 
another embodiment, processor 200 receives the data via multiplexer 206 and 
determines the field of view. Hther alternative is suitable for the present 
invention. 

Processor 200 is connected to memory 220 which stores the locations 
of the targets and images of the targets (or at least partial images). Memory 
220 also stores images of the replacement graphics, instructions for creating 
replacement graphics and/or instmctions for highUghting, editing, etc. Memory 
200 is loaded with its data and maintained by processor 222. The inventors 
contemplate that during operation of this system, processor 200 will be too busy 
to use compute time for loading and maintaining memory 220. Thus, a separate 
processor 222 is used to load and maintain the memory during operation. If 
cost is a factor, processor 222 can be eliminated and processor 200 will be used 
to load and maintain memory 220; however, for optimal performance memory 
220 should be loaded, if possible, prior to the broadcast. 

The images and locations of targets can be loaded into memory 220 
either manually or automatically. For example, if the target's image and 
location ate known in advance (e.g. an advertisement at the stadium) then prior 
to real-time operation of the system an operator can input the location of the 
target and scan in (or otherwise download) an image of the target. 
Alternatively, the operator can point one or more cameras at the target and use 
a mouse, light pen or other pointmg device to select the target's image for 
storing in memory 220. The location of the target can be determined by 
physical measurement, using pan/tUt/zoom sensors, etc. If the target is not 
known in advance (for example if the target is the first down yard line) then the 
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operator can select the target during operation using a pointing device and the 
system will download the image of the target and its location (using 
pan/tilt/zoom data) to memory 220. Alternatively, the system can be 
programmed to know that the target is one of a set of possible targets. For 
example, the system can be programmed to know that the target is a yard line 
and the operator need only input which yard line is the current target. The 
replacement gr£5)hics are loaded into memory after being digitized, downloaded 
or the replacement graphics can be created with processor 222. Instructions for 
highlighting or creating replacement graphics can be progranmied using 
processor 222 or processor 200. 

Processor 200 is connected to video modification unit 214. The output 
of video modification unit 214, labeled as signal 226, is the video signal 
intended for broadcast. This signal can be directly broadcast or sent to other 
hardware for further modification or recording. Video modification unit 214 
modifies the video signal from delay 210 with the data/signal from processor 
200. The type of modification can vary depending on the desued graphic 
result. One exemplar implementation uses a linear keyer as a video 
modification unit 214. When using a keyer, the signal fix)m the video processor 
200tothekeyerincludes two signals: YUV and an external key (alpha). The 
YUV signal is called foreground and the signal from delay 210 is called 
background. Based on the level of the extemal key, the keyer determines how 
much of the foreground and background to mix to determine the output signal, 
from 100 percent foreground and zero percent background to zero percent 
foreground and 100 percent background, on a pixel by pixel basis. 
Alternatively, video modification unit 214 can be another processor or video 
modification unit 214 can be a part of processor 200. 

In operation, processor 200 determines the field of view of the selected 
broadcast camera and checks memory 220 to see if any targets are within that 
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field of view. If so, processor 200 then determines the exact position of the 
target in a frame of video by determining which pixels represent the target. 
Processor 200 then checks memory 220 for the replacement graphic or 
instructions to make a replacement graphic (or highlight). If the replacement 
5 strategy is to highlight a certain portion of a field, then memory 220 may 
include instructions for changing the color of a certain portion of the field, 
shading of a certain portion of the field, etc. Based on the pan, tilt and zoom, 
and the actual image of the target, processor 200 determines the size and 
orientation of the replacement graphic (also called mapping). In one 
10 embodiment, the enhancement includes processor 200 creating a frame of video 
with a graphic at the position of the enhancement. The frame created by 
processor 200 is sent to video modification unit 214 which combines the frame 
from processor 200 with the frame fi*om delay 210. As will be described 
below, processor 200 is also used to account for occlusions. An alternate 
15 embodiment includes eliminating the sqjarate video modification unit and using 
processor 200 to edit the video signal from the selected broadcast camera. 

Figure 5 is a flow chart which explains the operation of the present 
invention. In step 300, video data is captured by a broadcast camera and is 
digitized. If the broadcast camera is a digital camera, digitizing is unnecessary, 
20 Simultaneously with step 300, pan, tilt and zoom data (field of view data) is 
sensed in step 302 and the field of view is determined in step 304. In step 306, 
processor 200 determines if any of the targets are within the field of view. 
Memory 200 (depicted in Figure 4) includes a database. In one alternative, the 
database stores the three dimensional locations of aU the targets. The field of 
25 view of a broadcast camera can be thought of as a pyramid whose location and 
dimensions are determined based on the field of view data. After determining 
the dimensions and locations of the pyramid, processor 200 accesses memory 
220 to determine if any of the targets are within the pyramid. Step 306 is a 
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quick method for determining if there is a target within the field of view of the 
camera. If not, the process is done and the system waits until the next frame 
of data. If there is a target within the field of view of the selected broadcast 
camera, then the exact position of the target must be determitied within the 
5 frame of video of the selected broadcast camera. 

Preferably, determining the position of the target is a two-step process. 
In the first step ( step 308) a rough estimate is made based on the pan, tilt and 
zoom values and in the second step the estimate of the target's position is 
refined (step 310). In regard to step 308, by knowing where the camera is 
10 pointed and the target's three dimensional location, the target's position in the 
video frame can be estimated. The accuracy of step 308 is determined by the 
accuracy of the pan/tflt/zoom sensors, the software used to determine the field 
of view and the stability of the platform on which the camera is located. In 
some alternatives, the field of view sensor equipment may be so accurate that 
15 the position of the target is adequately determined and step 310 is not necessary. 
In other instances, the pan, tilt and zoom data only provides a rough estunate 
308 (e.g a range of positions or general area of position) and stop 310 is needed 
to determine a more accurate position. 

Step 310 provides a more accurate determination of the target's position 
20 using pattern recognition techniques which are known in the art. Example of 
known pattern recognition and image processing technology can be found in the 
following documents: U.S. Patent No. 3,973,239, Pattern Preliminary 
Processing System; U.S. Patent No. 4,612,666, Automatic Pattern Recognition 
/^jpaiatus; U.S. Patent No. 4,674,125, Real-Time Hierarchal Pyramid Signal 
25 Processmg Apparatus; U.S. Patent No. 4,817,171, Pattem Recognition System; 
U.S. Patent No. 4,924,507, Real-Time Optical Multiple Object Recognition and 
Tracking System and Method; U.S. Patent No. 4,950,050, Optical Target 
Recognition System; U.S. Patent No. 4,995,090, Optoelectronic .Pattern 
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Comparison System; U.S, Patent No. 5,060,282, Optical Pattern Recognition 
Architecture Implementing The Mean-Square Error Correlation Algorithm; 
U.S. Patent No. 5,142,590, Pattern Recognition System; U.S. Patent No. 
5,241,616, Optical Pattern Recognitiori System Utilizing Resonator Array; U.S. 

5 Patent No. 5,274,716, Optical Pattern Recognition Apparatus; U.S. Patent No. 
5,465,308, Pattern Recognition System; U.S. Patent No. 5,469,512, Pattern 
Recognition Device; and U.S. Patent No, 5,524,065, Method and Apparatus 
For Pattern Recognition, It is contemplated that step 310 can use suitable 
technology other than pattern recognition technology. 

10 In step 312, processor 200 fetches the replacement graphic from 

memory 220. If memory 220 is storing instructions for replacement graphics, 
then processor 200 fetches the instructions and creates the graphic. For 
example, creating the graphic can include drawing a highlight for the yard line 
of a football field. In step 314, processor 200 determines the size and 

15 orientation of the replacement image, and maps the replacement image to the 
video frame. Memory 220 merely stores one size image. Because of the pan, 
tilt and zoom of the broadcast camera, the image stored in memory 220 may 
need to be mapped to the video ftame (e.g. magnified, reduced, twisted, 
angled, etc.). Processor 200 can determine the orientation based on the field 

20 of view data and/or the pattern recognition analysis in step 310. For example, 
by knowing where the broadcast camera is located and the pan, tilt and zoom 
of the broadcast camera, a computer can be programmed to figure how to map 
the replacement image or highlight on to the video frame. 

In step 316, the system accounts for occlusions. If there is an object or 

25 person in front of the target, then the enhanced video should show the object or 
person in fit>nt of the replacement graphic, highlight, etc. In one embodiment, 
the system cuts out a silhouette in the shape of the object or person from the 
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replacement image. Step 316 is discussed in morc detatt with req)^ 

In step 318, the system modifies the video of the original broadcast 
camera. As discussed above, this could include creating a second frame of 
video which includes a replacement image and using a keyer to combine the 

5 second frame of video with the original frame of video. Alternatively, a 
processor can be used to edit the frame of video of the broadcast camera. It is 
possible that within a given frame of video there may be more than one target. 
In that case steps 308-318 may be rq>eated for each target, or steps 308-316 
may be repeated for each target and step 318 be performed only once for all 

10 targets. Subsequmt to stq) 318, the enhanced frame of video may be broadcast 
or stored, and the process (steps 300-318) may repeat for another frame of 
video. 

Figure 6 is a more detailed flow diagram e3q>laining how the system 
accounts for occlusion. The stq)s described in Figure 6 are performed by a 

15 system which includes one or more dedicated cameras (e.g. dedicated camera 
142). Step 350, is perfonned before the live event occurs. In one embodunent, 
there is a dedicated camera substantially adjacent to a broadcast camera for each 
target that the broadcast camera may view. For example, if there are three 
advertisements which are to be replaced in the stadium and a particular camera 

20 can view two of those advertisements, then the system can include two 
dedicated cameras substantially adjacent to that particular camera. Prior to the 
game, a dedicated camera is pointed directly at one of the targets; the camera 
is zoomed in such that the target fills a substantial portion of the dedicated 
camera's frame of video; and the image of the target is stored in memory 220. 

25 A substantial portion means that the target typically appears to cover over half 
of the frame of video of the dedicated camera. For optimal results, the 
dedicated camera should be zoomed in such that the target fills the greatest 
amount of the frame of video possible while remaining completely within the 
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frame of video, unless it is desired to have clues of the scenery surrounding the 
target. After the dedicated camera is pointed at the target, its pan, tilt and 
zoom should remain fixed. 

Once the television broadcast of the live event begins, steps 352-362 are 
5 repeated for each frame where the occlusion analysis is desired. In step 352, 
a video image is captured and digitized by the dedicated camera. 
Simultaneously, a video image is captured by the broadcast camera. In step 
354, the digitized image from the dedicated camera is compared to the stored 
image of the target. The stored image is stored in memory 220. The processor 
10 knows which stored hnage to compare with from step 306 of Figure 5. The 
step of comparing could include altering one of the images such that both 
images are the same size and orientation, and then subtracting the data. 
Alternatively, other methods can be used to compare. If there is an occlusion 
blocking the target (step 356), then the two images will be significantly 
15 different and, in step 358, an occlusion will be reported. In rqxjrting the 
occlusion, the system reports the presence of an occlusion and the coordinates 
of the occlusion. When performing step 354, it is possible that there is no 
occlusion; however, the two images are not exactly the same. The differences 
between the images must meet a certain minimum threshold to be considered 
20 an occlusion. If the differences are not great enough to be an occlusion, then 
in step 360 the system determines that the differences are due to ambient 
conditions in the stadium. For example, if the lights have been dimmed then 
the c^tured image of the target may appear darker. Weather conditions could 
also have an effect on the appearance of the target image. If small differences 
25 are detected in step 360 that do not meet the threshold for occlusions, then the 
system "learns" the changes to the target by updating the stored image of the 
target to reflect the new lighting or weather conditions (step 362). For 
example, the new stored image of the target may be darker than the original 
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image. Subsequent to step 362 the system performs the rq)ort step 358 and 
reports that no occlusion was found. 

An alternative to the method of Figure 6 includes comparing the target 
image from the broadcast camera to the stored image. However, using the 
broadcast camera is not as advantageous as using a dedicated camera because 
it is likely that the broadcast camera would not be zoomed to the image. Thus, 
the target image is likely to be smaller on the broadcast camera than it will on 
the dedicated camera. Because there is a small image to work with, the system 
loses the subpixel accuracy obtained from the dedicated camera. Also, using 
a separate dedicated camera may increase the speed at which the system 
accounts for occlusions. 

Figure 7 shows an alternative embodiment of the present invention 
which utilizes electromagnetic transmittittg beacons at or near a target. The 
beacons transmit an electromagnetic signal not visible to the human eye. 
Electromagnetic waves include light, radio, x-rays, gamma rays, microwave, 
infrared, ultraviolet and others, all involving the propagation of electric and 
magnetic fields through space. The difference between the various types of 
electromagnetic waves are in the frequency or wave length. The human eye is 
sensitive to electromagnetic radiation of wave lengths from approximately 400- 
700iun, the range called light, visible light or the visible spectrum. Thus, the 
phrase "electromagnetic signal not visible to a human eye" means an 
electromagnetic wave outside of the visible spectrum. It is important that the 
signal transmitted from the beacon is not visible to human eye so that the visual 
appearance of the target will not be altered to those people attending the live 
event. In one embodiment, the beacon is an electromagnetic transmitter which 
includes infrared emitting diodes. Other sources which transmit 
electromagnetic waves may also used, for example, radio transmitters, radar 
repeaters, etc. 
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Figure 7 shows a broadcast camera 400 which outputs a video signal 
402. Broadcast camera 400 includes a zoom lens coupled to a zoom detector 
404. The output of zoom detector 404 is transmitted to analog to digital 
converter 406 which sends the digital output to processor 408 . Mounted on top 

5 of broadcast camera 400 is sensor 410. In the embodiment which utilizes an 
infrared emitter as a beacon, sensor 410 is an infrared sensor. Sensor 410 is 
mounted on top of broadcast camera 400 so that the optical axis of sensor 410 
is as close as possible to the optical axis of broadcast camera 400. It is also 
possible to locate sensor 410 near broadcast camera 400 and account for 

10 differences between optical axes using matrix transformations or other suitable 
mathematics. 

One example of an infrared sensor is a progressive scan, full frame 
shutter camera, for example, the TM-9701 by Puhiix. The Puhiix sensor is a 
high resolution 768(H) by 484(V) black and white full frame shutter camera 

15 witti asynchronous reset capability. The camera has an eight bit digital signal 
output and progressively scans 525 lines of video data, A narrow band infrared 
filter is affixed ia front of the lens of the Pulnix sensor. The purpose of the 
filter is to block electromagnetic signals that are outside the spectrum of the 
signal from the beacon. The sensor captures a frame of video (data) which 

20 comprises a set of pixels. Each pixel is assigned a coordinate corresponding to 
an X-axis and a y-axis. The sensor data includes an eight bit brightness value 
for each pixel, which are scaimed out pixel by pixel to interface 412 along with 
other timing information. Interface 412 ou^uts four signals: LDV, FDV, CK 
and DATA. LDV (line data valid) is transmitted to X-Y counters 414 and 

25 indicates that a new line of valid data is being scanned out of sensor 410. FDV 
(frame data valid) which is transmitted to X-Y counters 414 and memory 
control 416, indicates that valid data for the next frame is being transmitted. 
CK (pbcel clock) is a 14.318 MHZ clock from sensor 414 sent to X-Y counters 
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414 and memory control 416. X-Y counters 414 counts X and Y coordinates 
sequentially in order to keep track of the location of the pixel whose data is 
being scanned in at the current time. When LDV is inserted, the X counter is 
reset. When FDV is inserted, the Y counter is reset. 

The signal Data includes the eight bit data value for each pixel. As data 
is read from sensor 410, memory control 416 determines whether the pixels 
meets a brightness threshold. That is, noise and other sources will cause a large 
number of pixels to receive some data. However, the pixels receiving the 
signal from the beacon will have at least a minimum brightness level. This 
brightness threshold is set in a register (not shown) which can be set by 
processor 408. If the data for a particular pixel is above the brightness 
threshold, memory control 416 sends a write enable (WE) signal to memory 
418, causmg memory 418 to store the X and Y coordinates of the pixel, the 
data for that pixel and a code for that pixel. The code indicates that the data is 
valid data, a new frame, end of frame or a flash. Processor 408 can read the 
data from memory 418 and process the data locally or transmit the data to the 
production center (e.g., to multiplexer 206), 

Many arenas do not allow photographers to use flashes on their cameius 
in order to prevent impairing a player's vision from random flashes during a 
sporting event. In lieu of individual camera flashes, many arenas install a set 
of strobe flashes at or near the ceiling of the arenas and provide for 
communication between each photographer's camera and the set of strobe 
flashes. When the photographer takes a picture, the strobe flashes emit a flash 
of light, which may include an electromagnetic wave in the infrared spectrum. 
In one embodiment, the system avoids using incorrect data due to sensors 
detecting a flash by using filters. A second embodiment connects a signal from 
a strobe flash to a computer which causes the system to ignore data sensed 
during a flash. A third embodiment includes using flash detectors. The flash 
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detector can be located anywhere in the arena suitable for sensing a strobe flash. 
Figure 7 shows flash detector 422 which detects a flash and sends a signal to 
memory control 416. Flash detector 422 iacludes a photo detector which can 
comprise, at least, a photo diode and ah opamp. In front of the photo detector 
would be a filter that allows detection of signals in a spectrum that includes the 
signals emitted by the beacon. Connected to the opamp are components which 
can detect pulse edges. 

The embodiment described in Figure 7 operates similar to the 
embodiment described in Figure 3. Some of the differences between the 
operation of the two embodiments are depicted in Figure 8. Similar to the 
embodiment ia Figure 3, the embodiment in Figure 7 first captures and digitizes 
video data. In step 450, infrared data is received. In step 452, the system 
determines whether a target is found in the infrared data by monitoring the data 
stored in memory 418. Since memory control 416 only allows data above a 
threshold to be stored in memory 418, if a given frame of data from a sensor 
has pixel data stored in memory then a target is found. If a sensor is detectmg 
false targets, then various error correction methods known in the art can be 
utilized. In step 454, the position of the target is determined in the frame of 
video by reading the X and Y coordinates stored with the pixel data in memory 
418. Step 456 fine tunes the determined position information of the target to 
account for the error ftom the camera's platform or pan/tilt/zoom sensors. One 
alternative for accounting for the difference in optical axis is to use a 
transformation matrix; however, other mathematical solutions known in the art 
are also suitable. After step 456, the system can perform steps 312 through 318 
as described with respect to Figure 5, however, any field of view data used is 
based on the size and position of the beacon's signal in the sensor's frame of 
video. 
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A further alternative of Figure 7 includes using polarization. That is the 
infrared filter on sensor 410 is rq)laced or augmented with a polarized filter. 
A target to be replaced (e.g. , a billboard) is treated with a spectral coating that 
allows only polarized light to reflect off the billboard. The filter and spectral 
coating are designed such that light reflecting off the billboard to sensor 410 
will be completely blacked-out. The pixels that represent the position of the 
target in the sensor's frame of video will have a brightness value of zero or 
close to zero. Thus, memory control 416 is used to only store memoiy that has 
a brightness value of zero or below a threshold level. 

The foregoing detailed description of the invention has been presented 
for pxxrposes of illustration and description. It is not intended to be exhaustive 
or to limit the invention to the precise form disclosed, and obviously many 
modifications and variations are possible in light of the above teaching. The 
described embodiments of the system for enhancing the broadcast of a live event 
were chosen in order to best explain the principles of the invention and its 
practical application to therd)y enable others skilled in the art to best utilize the 
invention in various embodiments and with various modifications as are suited 
to the particular use contemplated. The invention is, thus, intended to be used 
with many different types of live events including various sporting events and 
nonsporting events. It is intended that the scope of the invention be defined by 
the claims api>ended hereto. 
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